Methods and apparatus for reducing resource contention in parallel data backup processes

ABSTRACT

Methods and apparatus for reducing resource contention in parallel data backup processes that share a variety of resources in a computer environment to access target data to be backed-up. The target data is distributed among the backup processes based on an availability of one or more resources in the computer environment used by a backup storage system to access the target data. In one example, each data file of the target data is assigned to a particular backup process based on the availability of one or more resources and a total size of the target data. The data files of the target data are assigned to the backup processes such that each backup process backs-up a respective portion of the target data, and such that the collection of backup processes that make up a backup operation are completed in a shortened time as a result of reduced resource contention. The target data originally may be located, for example, in one or more host computer systems, one or more primary storage systems, other computer devices containing a storage medium, or combinations thereof.

FIELD OF THE INVENTION

The present invention relates to data storage for computers, and moreparticularly, to methods and apparatus for optimizing data backupprocesses.

DISCUSSION OF THE RELATED ART

Virtually all computer applications rely on some form of data storage.The term “data” generally refers to any information, includingformatting information, computer code (executable programs), and otherinformation that may be manipulated and/or utilized by an executableprogram. Accordingly, data storage systems generally can be used, forexample, for storing computer code, a variety of variables and otherinformation, and particular arrangements and organizations of data, suchas databases.

In a variety of computer environments, a backup data storage system maybe desirable in addition to a primary data storage system. Generally, abackup data storage system provides redundant data storage and archivingto permit recovery of data in the event of, for example, corruption ofthe data in the primary storage system, failure of the primary storagesystem, and the like.

FIG. 1 illustrates one example of a conventional computer environment200 including a computer system 10, a primary storage system 12, and abackup storage system 14. In FIG. 1, the computer system 10 is coupledto the backup storage system 14 via line 18. The line 18 may representone or more direct connections dedicated exclusively to transmittinginformation between the computer system 10 and the backup storage system14. Alternatively, the line 18 may represent a non-dedicated connection,such as an open network connection (as indicated by the dashed arrows inFIG. 1), which can be used for other purposes. For example, when theline 18 is a network connection, it may also be connected to one or moreother computer systems similar to the computer system 10, or othernetwork devices (not shown). In a network environment, the computersystem 10 may be configured as a conventional network computer system.

In FIG. 1, the primary storage system 12 may be any type of storagesystem that includes one or more storage devices employing a storagemedia, such as magnetic or optical disks, magnetic tape, and the like.Additionally, the primary storage system may include a processor and oneor more controllers to manage writing data to, and reading data from,the storage media. While the primary storage system 12 is directlyconnected to the computer system 10 in FIG. 1, the primary storagesystem 12 alternatively may be coupled to the computer system 10 via anetwork (e.g., via the line 18 serving as a network connection or via adifferent network). One example of a storage system suitable for use asthe primary storage system 12 in FIG. 1 is the SYMMETRIX line of diskarrays, available from EMC Corporation of Hopkinton, Mass.

The backup storage system 14 of FIG. 1 may include one or more backupstorage devices (e.g., disk drives, tape storage, or any other storagemedia), together with one or more processors and/or controllers forstoring data on the backup storage media and recovering the data fromthe backup storage media. An example of a backup storage system suitablefor use as the backup storage system 14 is the EMC® Data Manager (EDM),also available from EMC Corporation, of Hopkinton, Mass.

The EDM backup system is a turnkey product designed to manage and storelarge amounts of backup data. In particular, the EDM backup system iscapable of providing unattended offline or online backup of one or morenetwork client computer systems. For example, in computer environmentssuch as that shown in FIG. 1 in which the line 18 represents a networkconnection, an EDM backup system may act as a backup server to one ormore client computer systems (such as the computer system 10) coupled tothe network. The EDM backup system is a fully integrated package thatincludes all required hardware (e.g., a host processor, controllers,user interface components, etc.), and software for storage mediamanagement and backup system operation. In addition to UNIX™environments, the EDM backup system fully supports, for example, OS/2™,NetWare™, Windows NT™ and other operating systems, and is capable ofperforming database backup and restore operations for vendors such asOracle™, Sybase®, Informix™, and Microsoft SQL™.

As shown in FIG. 1, the backup storage system 14 typically is coupled to(or includes) a user interface 22. The user interface 22 may include,for example, a keyboard, keypad and/or mouse, a display which is capableof displaying text and/or graphics, and a printer. The user interface 22may be integrated as part of the hardware and software package of thebackup storage system 14. Alternatively, some or all of the componentsconstituting the user interface may be discrete components coupled tothe backup storage system 14.

Through the user interface 22, a user (e.g., a system operator oradministrator) may initiate and/or schedule backup activities byinstructing the backup storage system to backup particular target datafiles at particular times (or, for example, periodically). Additionally,through the user interface, the user may access information related tobackup activities performed by the backup storage system 14. Forexample, via the user interface 22, a user may access (e.g., display orprint) information indicating the success or failure of completed backupdata storage operations performed by the backup storage system 14 on oneor more computer systems 10 coupled thereto. Information related to thebackup activities of the backup storage system 14 typically ismaintained in a portion of the storage area of the backup storagesystem.

As also shown in the computer environment of FIG. 1, the backup storagesystem 14 optionally may be directly connected to the primary storagesystem 12 via one or more dedicated lines 20 (e.g., an EDM backup systemmay be connected directly to a SYMMETRIX storage system via one or morededicated lines 20). For situations in which the line 18 represents anetwork connection, one or more dedicated lines 20 connecting the backupstorage system 14 to the primary storage system 12 may improveperformance when backing-up and restoring large amounts of data (e.g.,large databases) by avoiding the transfer of data over the network.

FIG. 2 is a more detailed illustration of a portion of FIG. 1, showingthe primary storage system 12 connected to the backup storage system 14via the one or more dedicated lines 20. FIG. 2 shows that a number ofparallel backup processes 24 ₁–24 _(N) may execute to backup to thebackup storage system 14, via the one or more lines 20, target data 26that is stored in the primary storage system 12. Each backup process 24₁–24 _(N) generally backs-up a respective portion of the target data 26to one or more storage media 28 of the backup storage system 14.

In FIG. 2, the backup processes 24 ₁–24 _(N) are shown as dashed lines,and the one or more dedicated lines 20 are shown for purposes ofillustration as a conduit between the primary storage system 12 and thebackup storage system 14, to clarify that each backup process 24 ₁–24_(N) represents a flow of data and not necessarily a particular discretephysical connection. More specifically, it should be appreciated thatthe number of backup processes 24 ₁–24 _(N) employed in theconfiguration of FIG. 2 is not necessarily related to the number ofdedicated lines 20 interconnecting the primary storage system 12 and thebackup storage system 14. For purposes of the present discussion, eachbackup process 24 ₁–24 _(N) merely requires some physical path betweenthe primary storage system 12 and the backup storage system 14, and aparticular physical path for a backup process may involve different onesof the dedicated lines 20 at different times. It is also possible thatdifferent backup processes may utilize one or more common dedicatedlines at different times during a backup operation of the target data.

In FIG. 2, it should also be appreciated that the block representationsof the target data 26 and the backup storage medium or media 28 areshown for purposes of illustration only. In particular, the target data26 may be located entirely on one storage medium in the primary storagesystem 12, or may be distributed amongst different storage media in theprimary storage system 12. Similarly, the backup processes 24 ₁–24 _(N)may backup the target data 26 entirely to one backup storage medium 28in the backup storage system 14, or may distribute the backed-up targetdata amongst different backup storage media 28 in the backup storagesystem 14.

In the data storage configuration of FIG. 2, typically a user specifies(e.g., via the user interface 22) the target data 26 to be backed up(e.g., by identifying one or more data files) and specifies the numberof parallel backup processes 24 ₁–24 _(N) that the backup storage system14 employs to backup the target data 26. In general, employing a greaternumber of parallel backup processes 24 ₁–24 _(N) tends to decrease thetime required to backup the target data 26. The number of backupprocesses that may be specified by a user theoretically is unlimited,but practically may be constrained by the particular design of thebackup storage system (e.g., software and/or hardware architecture).

Typically, in conventional implementations, the backup storage system 14of FIG. 2 obtains information regarding a total size of the target data26 to be backed up from the primary storage system 12, and assignsdifferent portions of the target data 26 to respective backup processes24 ₁–24 _(N) such that the target data is distributed essentiallyequally amongst the backup processes (i.e., each backup process backs-upapproximately the same amount of data). In particular, a desired size(e.g., in bytes) of the data to be backed-up by each process 24 ₁–24_(N) generally is determined from the total size of the target data 26to be backed up divided by the number of backup processes specified bythe user. For example, if the total size of the target data is tengigabytes and ten backup processes are specified, each backup processbacks-up approximately one gigabyte of data. Generally, a given datafile of the target data 26 is not broken up amongst different backupprocesses. Accordingly, in the above example, each backup process maynot backup exactly one gigabyte of data, as the distribution of wholedata files amongst the ten backup processes may result in slightlydifferent data sizes amongst the backup processes. In any case, itshould be appreciated from the foregoing description that, in theconventional data storage configuration shown in FIG. 2, generally thetotal size of the target data and the number of backup processes used tobackup the target data are the factors that determine how the targetdata is distributed amongst the backup processes.

FIG. 3 is a diagram showing a more detailed example of one possiblecomponent configuration for the primary data storage system 12 shown inFIGS. 1 and 2. The configuration shown in FIG. 3 is similar to that ofthe SYMMETRIX storage systems, discussed above. As shown in FIG. 3, theone or more dedicated lines 20 between the backup storage system 14 andthe primary storage system 12 may be coupled to the primary storagesystem via one or more front end controllers 40 ₁–40 _(N). Each frontend controller controls communication and data transfer to and from theprimary storage system 12 according to any one of a variety ofcommunication protocols. For example, any one of the front endcontrollers 40 ₁–40 _(N) may be a fiber channel adapter or a SCSI (SmallComponent System Interconnect) adapter employing SCSI communicationprotocol.

As shown in FIG. 3, the primary storage system 12 may include a numberof front end controllers 40 ₁–40 _(N), and each front end controller mayinclude a number of physically distinct communication ports to whichcommunication lines (e.g., the dedicated lines 20 or networkconnections) respectively are connected. For example, FIG. 3 shows thata first front end controller 40 ₁ accommodates two dedicated lines and asecond front end controller 40 ₂ accommodates four dedicated lines.Other front end controllers of the primary storage system 12 mayaccommodate different numbers of dedicated lines.

FIG. 3 also shows that the primary storage system 12 may include one ormore storage devices 46 ₁–46 _(N), shown for purposes of illustration inFIG. 3 as disk drives, or “disk drives.” It should be appreciated thatother types of storage devices may be suitable for use in the primarystorage system 12, as discussed above. Additionally, it should beappreciated that while five storage devices are shown in FIG. 3, theprimary storage system 12 may include any number of storage devices 46₁–46 _(N).

The primary storage system 12 of FIG. 3 also includes one or morestorage medium controllers 44 ₁–44 _(N) to control writing and readingof data to and from the one or more storage devices. In FIG. 3, thestorage medium controllers are shown for purposes of illustration asdisk controllers, wherein each disk controller is coupled to one or moredisk drives. FIG. 3 also shows that the disk controllers 44 ₁–44 _(N)and the front end controllers 40 ₁–40 _(N) are coupled to a cache memory42 that stores currently active data being written to or read from theprimary storage system 12.

In the primary storage system 12 of FIG. 3, a particular data file maybe stored entirely on one of the disk drives 46 ₁–46 _(N).Alternatively, a data file may include a number of segments, whereinsome of the segments may be stored on one disk drive while othersegments are stored on one or more other disk drives. Stateddifferently, one data file may include a number of segments stored in adispersed manner amongst a number of disk drives of the primary storagesystem 12.

In the example of the primary storage system 12 shown in FIG. 3, thereis not necessarily a one-to-one correspondence between the diskcontrollers 44 ₁–44 _(N) and the front end controllers 40 ₁–40 _(N), andthe primary storage system 12 may include different numbers of diskcontrollers and front end controllers. Additionally, the primary storagesystem 12 may be configured such that two or more disk controllers areassociated with a particular front end controller, while other front endcontrollers are associated with only one disk controller.

SUMMARY OF THE INVENTION

One embodiment of the invention is directed to an apparatus in acomputer environment including at least one data storage area and atleast one backup data storage system coupled to the at least one datastorage area. The backup storage system is configured to execute atleast two backup processes in parallel to backup target data stored inthe at least one storage area, wherein each backup process backs up arespective portion of the target data. The apparatus comprises at leastone contention controller, coupled to the backup storage system, thatdistributes the respective portions of the target data among the atleast two backup processes based, at least in part, on an availabilityof at least one resource in the computer environment used by the backupstorage system to access the target data.

Another embodiment of the invention is directed to an apparatus in acomputer environment including at least one data storage area and atleast one backup data storage system coupled to the at least one datastorage area. The backup storage system is configured to execute atleast two backup processes in parallel to backup target data stored inthe storage area. Each backup process backs up a respective portion ofthe target data. The apparatus comprises means, coupled to the at leastone backup storage system, for distributing the respective portions ofthe target data among the at least two backup processes based, at leastin part, on an availability of at least one resource in the computerenvironment used by the at least one backup storage system to access thetarget data.

Another embodiment of the invention is directed to a method in acomputer environment including at least one data storage area and atleast one backup data storage system coupled to the at least one datastorage area. The backup storage system is configured to execute atleast two backup processes in parallel to backup target data stored inthe storage area. Each backup process backs up a respective portion ofthe target data. The method comprises an act of distributing therespective portions of the target data among the at least two backupprocesses based, at least in part, on an availability of at least oneresource in the computer environment used by the at least one backupstorage system to access the target data.

Another embodiment of the invention is directed to a computer readablemedium encoded with a program for execution on a computer in a computerenvironment including at least one data storage area and at least onebackup data storage system coupled to the at least one data storagearea. The backup storage system is configured to execute at least twobackup processes in parallel to backup target data stored in the storagearea. Each backup process backs up a respective portion of the targetdata. The program, when executed on the computer, performs a methodcomprising an act of distributing the respective portions of the targetdata among the at least two backup processes based, at least in part, onan availability of at least one resource in the computer environmentused by the at least one backup storage system to access the targetdata.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a conventional computer environmentincluding a computer system, a primary data storage system, and a backupdata storage system;

FIG. 2 is a diagram showing the primary storage system and the backupstorage system of FIG. 1 in greater detail;

FIG. 3 is a diagram of one example of the primary storage system shownin FIGS. 1 and 2;

FIG. 4 is a diagram showing a computer environment in which paralleldata backup processes are executed based upon available resources in thecomputer system, according to one embodiment of the invention;

FIG. 5 is a diagram illustrating a flow chart of a method for reducingresource contention amongst parallel data backup processes, according toone embodiment of the invention;

FIGS. 6A and 6B are diagrams illustrating a flow chart of a method forreducing resource contention amongst parallel data backup processes,according to another embodiment of the invention;

FIG. 7 is a graph illustrating a status of parallel data backupprocesses during a data file pre-assignment procedure for reducingresource contention, according to one embodiment of the invention;

FIG. 8 is a diagram showing an example of a data file resourceinformation table for one data file of target data to be backed-up,according to one embodiment of the invention;

FIG. 9 is a diagram showing an example of a backup process resourceinformation table for one backup process, according to one embodiment ofthe invention;

FIG. 10 is a diagram showing an example of a contention penalty tablefor one data file of the target data, according to one embodiment of theinvention;

FIG. 11 is a diagram showing an example of a backup process executiontable, according to one embodiment of the invention; and

FIG. 12 is a graph illustrating the concept of guard bands for resourcecontention analysis, according to one embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

As discussed above in connection with FIG. 2, a conventional backupstorage system in a computer environment may perform a data backupoperation by executing a number of parallel (i.e., concurrent) backupprocesses to backup target data that is stored somewhere in the computerenvironment. For example, FIG. 2 shows that the target data to bebacked-up may be stored in one or more primary storage systems dedicatedto data storage. Similarly, it should be appreciated that all or aportion of the target data to be backed-up may be stored in one or morecomputer systems coupled to the backup storage system either via adedicated (i.e., direct) interface or via a non-dedicated interface suchas a network connection, as shown for example in FIG. 1.

In any case, regardless of the original location of the target data,typically the significant factors that determine how the target data isdistributed amongst multiple parallel backup processes of a conventionalbackup storage system during a data backup operation include only thetotal size of the target data and the number of backup processes used tobackup the target data, as discussed above in connection with FIG. 2. Inparticular, the target data generally is distributed essentially equallyamongst the backup processes, such that each backup process backs-upapproximately the same amount of data, and such that no data files arebroken up amongst different backup processes.

Applicants have appreciated that in the conventional parallel backupprocess scheme described above, the backup storage system may use avariety of resources in the computer environment to access the targetdata during a given backup operation. For example, in accessing targetdata stored in a primary storage system similar to the primary storagesystem 12 shown in FIG. 3, the backup storage system may require theavailability of a particular front end controller 40 ₁–40 _(N), aparticular disk controller 44 ₁–44 _(N), and a particular disk drive 46₁–46 _(N) at a particular time to access a portion of the target data.Accordingly, the disk drives 46 ₁–46 _(N), the disk controllers 44 ₁–44_(N), and the front end controllers 40 ₁–40 _(N) are resources of thestorage system 12 through which one or more segments of a data filestored in the primary storage system 12 are accessed.

Based on the configuration of the primary storage system 12 shown inFIG. 3, namely the allocation of these resources, generally there is aparticular access path through the primary storage system 12 to accesseach segment of a data file. For example, as shown in FIG. 3, disk drive46 ₁ is coupled to disk controller 44 ₁. The primary storage system 12may be configured such that the disk controller 44 ₁ transports data toand from the primary storage system 12 via front end controller 40 ₂.Accordingly, a segment of a data file stored on the disk drive 46 ₁ hasan access path that makes use of the combination of resources includingthe front end controller 40 ₂, the disk controller 44 ₁, and the diskdrive 46 ₁. Stated differently, each segment of a data file stored inthe primary storage system of FIG. 3 may be identified in terms of theresources utilized to access the segment, which in the foregoing examplemay include a front end controller identifier, a disk controlleridentifier, and a disk drive identifier.

Applicants have recognized that, with respect to resources in general,it is possible that two or more backup processes may attempt to accesssome of the same resources in the computer environment at the same timeduring a backup operation, resulting in resource contention that reducesthe performance of the backup operation. For example, with referenceagain to the example of a primary storage system shown in FIG. 3, it ispossible that at a particular time during a backup operation, two backupprocess may attempt to access different respective data files that arelocated on the same disk drive of the primary storage system. Since onlyone data file may be accessed from the disk drive at any given time, onebackup process must wait for the other to access its respective file.For purposes of the present discussion, this situation may be termed“disk drive contention.”

In a similar manner, even though two or more different backup processesmay be attempting to access data files on different disk drives at aparticular time during a backup operation, the primary storage system ofFIG. 3 may be configured such that the backup processes must use thesame front end controller 40 ₁–40 _(N) and/or the same disk controller44 ₁–44 _(N) to get to the different disk drives. Hence, this situationresults in one or both of “front end controller contention” and “diskcontroller contention” between two or more backup processes.

It should be appreciated that the resources identified above inconnection with the primary storage system shown in FIG. 3 are forpurposes of illustration only; in particular, based on the originallocation of the target data in the computer environment (e.g., in one ormore host computers and/or primary storage systems coupled directlyand/or via a network to the backup storage system), the backup storagesystem may require the availability of a wide variety of resources inthe computer environment from time to time to access the target dataduring a backup operation. These resources similarly may provide forsituations involving resource contention. In general, the aspects of theinvention discussed further below may be used with any number ofdifferent resources and configurations of resources, and are not limitedto use with any particular types and/or configurations of resources.

Applicants have recognized that the potential for resource contentionmay undesirably prolong a backup operation, as one or more backupprocesses may sit idle waiting for one or more resources to becomeavailable. As discussed above, present procedures for executing parallelbackup processes typically consider only the size of the target data andthe number of backup processes, without regard to the resources thatmust be used to access the data (e.g., where the data is physicallylocated, what physical and/or logical paths are available to get to thedata, etc.). Applicants have realized that by considering theavailability of resources used by the backup storage system to accessthe target data, and distributing the target data amongst the parallelbackup processes based at least in part on the availability of theseresources at given points in time, resource contention can be reducedduring a backup operation, thereby improving the performance of theoperation. Stated differently, Applicants have appreciated that byintelligently allocating resources to parallel backup processes so as tominimize contention between the resources, the collection of backupprocesses that make up the backup operation may be completed in ashorter time.

Accordingly, one embodiment of the present invention is directed toreducing resource contention in parallel data backup processes thatshare a variety of resources in a computer environment to access targetdata to be backed-up. In this embodiment, the data to be backed up isdistributed among the backup processes based, at least in part, on anavailability of one or more resources in the computer environment usedby a backup storage system to access the target data.

In one aspect of this embodiment, each data file of the target data isassigned to a particular backup process based on the availability of oneor more resources. The data files of the target data are assigned to thebackup processes such that each backup process backs-up a respectiveportion of the target data, and such that the collection of backupprocesses that make up the backup operation are completed in a shortenedtime as a result of reduced resource contention.

The present invention, in various embodiments, may be employed toback-up target data located in any number and variety of storage areas,as the invention is not limited to backing up target data from anyparticular place. For example, the target data originally may be locatedin one or more host computer systems, one or more primary storagesystems, other storage devices, or a combination thereof. Accordingly,for purposes of the following discussion, the original location of thetarget data is referred to generally as a “storage area,” whether thisstorage area be part of a host computer, primary storage system, orother device in the computer environment containing a storage medium.

It should be appreciated that the embodiments of the invention outlinedin further detail below in connection with methods and apparatus forreducing resource contention in parallel data backup processes may beimplemented in any of numerous ways, as the invention is not limited toany particular manner of implementation. Examples of specificimplementations are provided for illustrative purposes only.

FIG. 4 shows a computer environment similar to that shown in FIG. 2,including a primary storage system 12A, a backup storage system 14Acoupled to the primary storage system via one or more dedicated lines20, and a contention controller 34 according to one embodiment of theinvention. The contention controller 34 controls one or more backupprocesses 24 ₁–24 _(N) based at least in part on resource information 30regarding the availability of one or more resources used by the backupstorage system 14A to access the target data 26. In one aspect, thecontention controller 34 facilitates an efficient data backup operation,wherein the backup operation as a whole is completed in a shortened timeas a result of reduced resource contention.

As shown in the example of FIG. 4, in one embodiment, the contentioncontroller 34 obtains the resource information 30 from the primarystorage system 12A in which the target data 26 is stored. However, itshould be appreciated that the invention is not limited in this respect,as the backup storage system 14A may backup target data from a varietyof data storage areas using a variety of different resources in avariety of different computer environments, as discussed above.Accordingly, for purposes of the invention, information regarding theavailability of resources used by the backup storage system 14A toaccess the target data may be made available to the backup storagesystem 14A in a variety of ways. Additionally, according to anotherembodiment of the invention, the backup storage system 14A need notbackup target data and/or obtain resource information via the dedicatedlines 20, but may do so via a network connection.

Furthermore, although the contention controller 34 is shown in FIG. 4 asbeing included in the backup storage system 14A for purposes ofillustration, it should be appreciated that the invention is not limitedin this respect. The contention controller 34 may be implemented usinghardware, software, firmware, or a combination of the foregoing.Moreover, rather than being implemented in the backup storage system14A, the contention controller 34 alternatively may be a stand alonehardware implementation, a software implementation in a dedicatedcomputer or processor (e.g., a server) coupled to the backup storagesystem 14A, may be included in the primary storage system 12A or one ormore host computer systems coupled to the backup storage system (e.g.,as shown in FIG. 1), or may be distributed among two or more systemcomponents. According to various aspects of the invention, thecontention controller 34 generally is coupled to at least the backupstorage system 14A and is capable of accessing information regarding theavailability of resources in the computer environment used to backup thetarget data.

FIG. 5 is a diagram showing a flow chart of a method for reducingresource contention amongst parallel data backup processes, according toone embodiment of the invention. In one aspect of the invention, thecontention controller 34 of FIG. 4 implements the method outlined inFIG. 5. In this aspect, according to the method of FIG. 5, thecontention controller 34 of FIG. 4 processes each data file of thetarget data to be backed up, and first pre-assigns the respective datafile to each backup process of the backup operation to assess the degreeof resource contention that would result from such an assignment.Subsequently, based on an assessment of potential resource contentionfor each pre-assigned backup process, the contention controller actuallyassigns each data file to the particular backup process for which thelowest degree of resource contention results.

In particular, according to the method of FIG. 5, in step 49 a firstcandidate data file of the target data is selected, and in step 50 thecandidate data file is pre-assigned to a first backup process of two ormore parallel backup processes. In step 52, the method compares theresources that would be used by the first backup process to backup thecandidate data file with the resources that concurrently would be usedin all other backup processes to backup other data files of the targetdata previously assigned to these other backup processes. In step 54, acontention penalty is calculated for the candidate data file in thepre-assigned backup process (i.e., the first process in this example)based on any identical resources that concurrently would be used both bythe backup process to which the candidate data file is pre-assigned andthe other backup processes. It should be appreciated that for a firstcandidate data file of the target data (i.e., no other data files of thetarget data previously have been assigned to any of the backupprocesses), there is no contention penalty.

As shown in steps 56 and 58 of FIG. 5, the method subsequentlypre-assigns the candidate data file consecutively to each other backupprocess of the backup operation, and with each pre-assignment similarlycompares resources in step 52 and calculates a contention penalty instep 54 for the candidate data file in the pre-assigned backup process.Once a contention penalty for the candidate data file is obtained foreach pre-assigned backup process, the candidate data file is actuallyassigned to the backup process which resulted in the lowest contentionpenalty during the pre-assignment procedure, as shown in step 60.

Once the candidate data file is assigned to a particular backup processin step 60, the method of FIG. 5 proceeds through steps 62 and 64 tochoose new candidate data files from the target data, and the foregoingpre-assignment and actual assignment procedures are repeated for eachnew candidate data file until all data files of the target data havebeen assigned to a particular backup process. In one aspect, as a resultof performing the method of FIG. 5, a table or executable file may becompiled, in which is listed an identifier (e.g., a name) for each datafile, the backup process to which it is assigned, and its particularorder of consideration during the backup process to which it isassigned. One example of such a table is shown for purposes ofillustration in FIG. 11, which is discussed in greater detail below. Inother aspects, depending on the specific configuration of the backupstorage system 14A and the contention controller 34 shown in FIG. 4, thebackup storage system or the contention controller may execute such afile (or, for example, consult such a table) to initiate and perform thebackup operation.

In the embodiment shown in FIG. 5, each data file of the target data isprocessed (i.e., pre-assigned to each backup process and actuallyassigned to one particular backup process) only once by the method tocompile, for example, an executable file or table similar to that shownin FIG. 11. As a result, processing time for the method is minimized,which is advantageous when the method is performed as part of the backupoperation. However, it should be appreciated that the invention is notlimited in this respect. For example, according to another embodiment,the method of FIG. 5 may be iterated a number of times using differentorders of consideration for the candidate data files of the target datato compile a number of tables similar to FIG. 11, each having aparticular arrangement of target data file assignments among the backupprocesses. Once a number of such tables are compiled, the method maydetermine if one particular arrangement of target data file assignmentsresults in a shortest completion time for a backup operation. In thismanner, the method of FIG. 5 may be iterated a number of times tooptimize the assignment of target data files to backup processes so asto insure a shortest completion time for the backup operation.

In the method of FIG. 5, the data files that make up the target data maybe processed in any order. In one embodiment, the candidate data filesof the target data are processed in a particular order from largest tosmallest. For example, according to this embodiment, in the step 49 ofFIG. 5 a first candidate data file having the largest size amongst allof the data files of the target data is selected, and with eachiteration of the step 64 a next smaller data file of the target data isselected as the new candidate data file. In this manner, the methodaccording to this embodiment attempts to reduce resource contentionamongst those data files of the target data that generally would requirethe longest time to back-up (due to their relatively larger sizes) byassigning them to backup processes before smaller data files of thetarget data.

While not shown explicitly in the method of FIG. 5, alternative methodsfor reducing resource contention according to yet other embodiments(discussed in detail further below) may consider additional factorsother than the lowest contention penalty in the actual assignment of thecandidate data file to a particular backup process (i.e., in step 60 ofFIG. 5). For example, such alternative methods may consider if more thanone backup process resulted in the same lowest contention penalty forthe candidate file. Additionally, such methods may consider relativesizes of the data previously assigned to each backup process, so as tomaintain a balance amongst the data loads of the backup processes;stated differently, such methods may actually assign the candidate datafile to a particular backup process based not only on resourcecontention factors, but based on maintaining approximately similar sizesof the total data assigned to each backup process.

In this respect, it should be appreciated that according to the methodof FIG. 5, the backup process resulting in the lowest contention penaltyduring the pre-assignment procedure generally is the backup process towhich the most data previously has been assigned (i.e., the largestbackup process). This may be more clearly understood by noting that, atthe point at which a candidate data file is pre-assigned to the largestbackup process, no other smaller backup processes are using resourcesbecause no data files have yet been assigned to them; hence, there is noresource contention. This situation may result in an undesirableimbalance in the data loads among the backup processes.

Accordingly, as discussed above, methods according to other embodimentsof the invention may employ various procedures to ensure that data loadsamong the backup processes are approximately balanced. This may beaccomplished in any of numerous ways, as the present invention is notlimited to any particular technique. For example, in one embodiment therelative sizes of the total data assigned to each process is consideredbefore assignment of a new candidate data file to a particular process,as discussed above.

Following below is a more detailed discussion of another method forreducing resource contention similar to the method of FIG. 5, accordingto one embodiment of the invention. In the following discussion, withreference again to FIG. 4, the example of a storage area for the targetdata 26 is considered to be similarly configured to the primary storagesystem 12 shown in FIG. 3. Of course, it is to be understood that theinvention is not limited to use in backing up target data stored in thisparticular configuration of a storage area. As discussed above inconnection with the exemplary primary storage system of FIG. 3, thetarget data 26 may be stored on one or more disk drives 46 ₁–46 _(N). Avariety of access paths through the primary storage system to the diskdrives 46 ₁–46 _(N) may be determined by the front-end controllers 40₁–40 _(N) and the disk controllers 44 ₁–44 _(N).

Based on the resources used to access target data in this particularexample of a primary storage area, Applicants have identified at leastthree potential sources of resource contention; namely, disk drivecontention, disk controller contention, and front-end controllercontention, as discussed above. The penalty incurred for contention maynot be the same for each resource. For example, Applicants haveidentified that in a primary storage system similar to that shown inFIG. 3, the penalty incurred for disk drive contention exceeds that fordisk controller contention and front-end controller contention. Thus, inone embodiment for use with such systems, priority is given to reducingdisk drive contention. It should be appreciated that in otherembodiments of the invention, different priorities may be given todifferent resources depending on various factors that may affect theoverall backup operation (e.g., the likelihood that a given resource maycontribute to performance degradation).

In view of the foregoing, according to one embodiment, methods andapparatus of the invention take into account both the physical locationof the target data 26, as well as the one or more access paths to thephysical locations in which the target data is stored, to assess theavailability of resources used to access the target data. In particular,according to one embodiment, the contention controller 34 of FIG. 4(and, similarly, a method for reducing resource contention discussedbelow in connection with FIGS. 6A and 6B) considers each of theforegoing potential sources of resource contention (i.e., disk drivecontention, disk controller contention, and front-end controllercontention) to ensure that the total backup operation executes asquickly as possible.

FIGS. 6A and 6B are diagrams illustrating a flow chart of a method forreducing resource contention amongst parallel data backup processes,according to one embodiment of the invention. The method of FIGS. 6A and6B is similar to the method outlined in FIG. 5, but is explained in moredetail and is more tailored to backing up target data that is stored ina primary storage system 12A similar to that shown in FIG. 3. As in themethod of FIG. 5, the method of FIGS. 6A and 6B employs a data filepre-assignment and actual assignment procedure which assigns each datafile of the target data to a particular backup process based on anavailability of resources. In addition, to keep the lengths of thebackup processes approximately equal, the method also considers therelative data sizes of each backup process in assigning data files tobackup processes. In one aspect, by implementing the method of FIGS. 6Aand 6B, the contention controller 34 of FIG. 4 creates a backup processexecution table or file (as shown in FIG. 11 and discussed in furtherdetail below) which is executed to control the backup processes of abackup operation performed by the backup storage system 14A.

FIG. 7 is a diagram illustrating a resource allocation of parallel databackup processes as a function of time during a data file pre-assignmentprocedure for reducing resource contention (e.g., as discussed above inconnection with FIG. 5). The graph of FIG. 7 is helpful to explain themethod outlined in FIGS. 6A and 6B, and also applies to the methodoutlined in FIG. 5. In the graph of FIG. 7, the left and right verticalaxes represent elapsed time t, and each backup process 24 ₁–24 _(N) issymbolically illustrated as a bar graph at a respective position alongthe horizontal axis, indicated by a backup process (BP) number (i.e., 1,2, . . . N). For purposes of clarity in the following discussion, thebackup processes are referred to as BP₁–BP_(N) rather than by theirreference characters 24 ₁–24 _(N).

The graph of FIG. 7 essentially represents a “snapshot” view of portionsof the target data (in terms of data files) that have been formallyassigned to each backup process BP₁–BP_(N) at some point during the datafile pre-assignment and actual assignment procedure, as discussed abovein connection with FIG. 5 and further below in connection with FIGS. 6Aand 6B. In particular, in the graph of FIG. 7, the shaded portions 26₁–26 _(N) of each bar graph represent data actually assigned to eachbackup process at the time of the snapshot.

As indicated in FIG. 7, according to one embodiment of the invention,all backup processes BP₁–BP_(N) essentially are initiated (e.g., by thecontention controller 34) simultaneously at time t=0. Additionally,according to one embodiment, each backup process is assumed to backup arespective portion of the target data at approximately a same data rater (e.g., in bits per second, or bps). In this manner, the amount of data26 ₁–26 _(N) assigned to a particular backup process is directly relatedto an elapsed time for that process by the data rate r. For example, ifthe data assigned to a particular backup process has a size b (e.g., inbytes), an elapsed time t_(BP) for the backup process may be determinedby:t _(BP)(sec)=b(bytes)×8(bits/byte)/r(bps).As shown in FIG. 7, at the time of the snapshot, the backup process BP₁has an elapsed time of t₁(as indicated on the left vertical axis) andthe backup process BP_(N) has an elapsed time of t_(N) (as indicated onthe right vertical axis).

With reference again to both FIGS. 4 and 6A, in step 70 of the methodaccording to the embodiment of FIGS. 6A and 6B, a user specifies (e.g.,via the user interface 22 shown in FIG. 4) the target data filesF_(1, 2, 3), . . . and a number N of backup processes that the backupstorage system 14A uses to backup the target data 26 during a backupoperation. Once the target data files are specified, information isobtained (e.g., from the primary storage system 12A) regarding a size ofeach of the data files. According to one aspect of this embodiment, asshown in step 72 of FIG. 6A, based on such information the target datafiles are arranged in a list, in order from the largest sized file(F_(largest)) to the smallest sized file (F_(smallest)). By ordering thetarget data files from largest to smallest, the method of FIGS. 6A and6B first processes (i.e., pre-assigns and actually assigns) those datafiles that may be more likely to contribute to significant resourcecontention due to their relatively larger sizes, and hence, relativelylonger backup times (the backup time t_(F) of a data file is related toits size by the data rate r). It should be appreciated, however, thatthe invention is not limited in this respect, as the data filesF_(1, 2, 3), . . . may be arranged in a variety of different orders.

Each data file F of the target data may comprise a number M of datasegments S. In particular, some segments of a given data file may bestored in different physical locations within a storage area, and accessto each segment of the file may require the availability of one or moreparticular resources that create an access path to the segment. Morespecifically, in the current example, different data segments of a givendata file may be located on different disk drives of the primary storagesystem 12A, which disk drives respectively may be accessed viaparticular combinations of front-end controllers and disk controllers.

Accordingly, as shown in step 74 of FIG. 6A, resource informationidentifying those resources that are used to access each segmentS₁–S_(M) of each data file F of the target data is collected andassembled in a table, for example, as shown in FIG. 8. It should beappreciated that, according to one embodiment, the method of FIGS. 6Aand 6B maintains a table similar to that shown in FIG. 8 for each datafile F of the target data.

In the data file resource information table of FIG. 8, each of the rowsrepresents one data segment S of the data file F, and may identify thesegment in terms of a time required, or duration D, to back up thesegment based on the size of the segment (e.g., in bytes) and the datarate r (e.g., in bps) assumed for each backup process. Accordingly, theduration D of each segment may be calculated in a manner similar to thatdiscussed above in connection with an elapsed time t_(BP) for eachbackup process, based on data size and data rate. For example, in FIG.8, the segment S₁ is indicated as having a duration D₁ =10 ms, and thesegment S₂ is indicated as having a duration D₂=20 ms. Other segmentsS₃–S_(M) are indicated by the durations D₃–D_(M), respectively. Based onthe foregoing, it should be appreciated that a total backup time t_(F)for the file F may be calculated based on a sum of all of the segmentdurations D₁+D₂+ . . . +D_(M) or, alternatively, from a sum of all ofthe segment sizes S₁+S₂+ . . . +S_(M) multiplied by the data rater.

In the data file resource information table shown in FIG. 8, each columnrepresents a particular resource in the computer environment that may beused to access the segment S of the data file F. In particular, FIG. 8shows that a first column indicates a first resource identifier R₁,which in the current example corresponds to a disk drive identifier.Similarly, a second column indicates a second resource identifier R₂,which in the current example corresponds to a disk controlleridentifier, and a third column indicates a third resource identifier R₃,which in the current example corresponds to a front-end controlleridentifier. It should be appreciated that while three columns are shownin the table of FIG. 8 for this particular example using threeresources, the table of FIG. 8 may include any number of columns, eachcolumn corresponding to a respective resource in the computerenvironment used to access the target data.

In the exemplary data file resource information table shown in FIG. 8,for the data segment S₁, the disk drive identifier “2” is indicated inthe first column to identify one of the disk drives 46 ₁–46 _(N) of theprimary storage system 12 shown in FIG. 3. Similarly, the diskcontroller identifier “3” is indicated in the second column to identifyone of the disk controllers 44 ₁–44 _(N), and the front-end controlleridentifier “1” is indicated in the third column to identify one of thefront-end controllers 40 ₁–40 _(N). These identifiers are selected forpurposes of illustration only, and a variety of identifiers may be usedto identify particular resources in the computer environment. Similarly,for the data segment S₂, the disk drive identifier “6” is indicated inthe first column to identify one of the disk drives 46 ₁–46 _(N) of theprimary storage system 12 shown in FIG. 3, the disk controlleridentifier “2” is indicated in the second column to identify one of thedisk controllers 44 ₁–44 _(N), and the front-end controller identifier“2” is indicated in the third column to identify one of the front-endcontrollers 40 ₁–40 _(N).

It should be appreciated that the foregoing example of a data fileresource information table as shown in FIG. 8 is provided for purposesof illustration only, and that the invention is not limited to thisparticular example. Other manners of compiling and maintaining data fileresource information are possible according to other embodiments of theinvention. Additionally, rather than compiling resource information on afile-to-file basis, resource information alternatively may be compiledbased on different data units or groupings, as the invention is notlimited to compiling and maintaining this information on a file-to-filebasis.

Once the method of FIGS. 6A and 6B has assembled a table similar to thatshown in FIG. 8 for each data file F of the target data (according tothe step 74), the method sets a file pointer to the largest data fileF_(largest), as indicated in step 76. As discussed above, it should beappreciated that according to other embodiments, the method initiallymay set the file pointer to a data file of the target data other thanF_(largest), according to a different ordering of the files. Aftersetting the file pointer, the method subsequently initiates a data filepre-assignment procedure, as indicated in step 78. For purposes of thepresent discussion however, it is more instructive to explain thepre-assignment procedure outlined in FIG. 6A not by starting with thefirst file processed (e.g., the largest file F_(largest)), but rather ata later point during the pre-assignment procedure, after which a numberof data files already have been actually assigned to a number ofrespective backup processes (as shown for example in the “snapshot”graph of FIG. 7). Accordingly, the following discussion describes thepre-assignment procedure of FIG. 6A as if the step 78 were being enteredinto not from the step 76, but rather from the link 122, at which pointa candidate file F other than the first file being processed (e.g., thelargest data file F_(largest)) and the last file being processed (e.g.,the smallest data file F_(smallest)) is being considered.

With reference again to FIG. 7, the candidate file F being pre-assignedto the backup process BP₁ in the step 78 of FIG. 6A (i.e., BP=1) has twosegments S₁ and S₂, each segment having a respective duration D₁ and D₂(i.e., the number of segments M in the candidate file F equals two). Itshould be appreciated that a candidate file F having only two segmentsis selected for purposes of illustration only, and that files of thetarget data may have any number M of data segments S. As discussed abovein connection with FIG. 8, each segment S₁ and S₂ of the candidate fileF is accessed via particular resources R₁, R₂, and R₃, which in thepresent example include a particular disk drive, a particular diskcontroller, and a particular front-end controller, respectively, of theprimary storage system 12A of FIG. 4.

In step 80 of FIG. 6A, a time pointer t_(p) is set to an elapsed time ofthe backup process BP₁ when the candidate file F is pre-assigned to it,which is indicated as a time t₁ on the left vertical axis of FIG. 7.Also indicated on the left vertical axis of FIG. 7 is a time (t₁+D₁),which corresponds to an elapsed time for the backup process BP₁ afterthe segment S₁ would be backed up, as well as a time (t₁+D₁+D₂), whichcorresponds to an elapsed time for the backup process BP₁ after thesegment S₂ (i.e., entire file F) would be backed up.

In step 82 of FIG. 6A, the method compares the resources R₁, R₂, and R₃used to access the segment S₁ of the candidate file F (as indicated inthe first row of the table shown in FIG. 8) with resources used in allother backup processes during the time interval between t₁ and (t₁+D₁).With reference again to FIG. 7, the interval between t₁ and (t₁+D₁) isindicated by two horizontal dashed lines, which identify a correspondingtime interval in each of the other backup processes BP₂–BP_(N). Duringthe time interval between t₁ and (t₁+D₁), FIG. 7 shows that some datafiles of the target data previously have been assigned to at least thebackup processes BP₂ and BP_(N) (as indicated by the shaded areas 26 ₂and 26 _(N), respectively). Accordingly, various resources are utilizedby at least the backup processes BP₂ and BP_(N) to access the assigneddata files during this interval. Some of these resources may be the sameas the resources used to access the segment S₁ of candidate file F,resulting in potential resource contention.

According to one aspect of the method of FIGS. 6A and 6B, a backupprocess resource information table is maintained for each backup processBP₁–BP_(N), an example of which is shown in FIG. 9. As with the table ofFIG. 8, it should be appreciated that the table in FIG. 9 is shown forpurposes of illustration only, in that other manners of compiling andmaintaining backup process resource information are possible, so thatthe present invention is not limited to any particular compilationtechnique. The table of FIG. 9 includes one column for each resource, ina manner similar to that of the data file resource information tableshown in FIG. 8, and one row for every data segment assigned to theparticular backup process. Each row of the table of FIG. 9 indicates anelapsed time, from the start of all of the processes at t=0, in terms ofthe durations of the respective data segments assigned to the backupprocess. It should be appreciated in FIG. 9 that the duration notationD₁–D_(M) is used generically to indicate the elapsed time of a processin terms of the durations of the segments assigned to the process, andthat the notation in FIG. 9 does not refer to the specific durations D₁and D₂ of the particular file segments S₁ and S₂ of the candidate file Fshown in FIG. 7. Accordingly, as illustrated in FIG. 7, the resourcesused by each backup process BP₂–BP_(N) during the interval between t₁and (t₁–D₁) (corresponding to the pre-assignment of segment S₁ of thecandidate file F to the backup process BP₁) may be identified byexamining the corresponding time interval in the table of FIG. 9 foreach backup process.

In this manner, in step 82 of FIG. 6A, the method compares the resourcesindicated in the first row of the table of FIG. 8 for the segment S₁ ofthe candidate file F with the resources used in a row of the table ofFIG. 9 corresponding to the time interval between t₁ and (t₁–D₁) foreach backup process BP₂–BP_(N). For each resource R₁, R₂, and R₃identified for the segment S₁, the method adds the total cumulativeamount of time that the resource is used in any of the other backupprocesses BP₂–BP_(N) during the time interval between t₁ and (t₁+D₁) togenerate a “segment penalty” (e.g., in an appropriate unit of time suchas milliseconds) for the resource. For example, with reference again toFIG. 7, if during the entire time interval between t₁ and (t₁+D₁) thebackup process BP₂ and the backup process BP_(N) both use the sameresources R₁, R₂, and R₃ that are used to access the segment S₁, thesegment penalty for each resource in units of time would be 2D₁ (i.e.,there would be three segment penalties for the segment S₁, one for eachresource R₁, R₂, and R₃, and each having a value in units of time of2D₁).

As shown in step 84 of FIG. 6A, the step 82 is repeated for each of thesegments S₂–S_(M) of the candidate file F (in the current example thereis only S₂). With reference again to the graph of FIG. 7, during thetime interval between (t₁+D₁) and (t₁+D₁+D₂), the backup process BP₂ hasbeen actually assigned only a small portion of data, and the backupprocess BP_(N) has been actually assigned some data, but not enough tocompletely fill this interval. Accordingly, the segment penalties forthe segment S₂ for each resource R₁, R₂, and R₃ likely will be less thanthe corresponding segment penalties for segment S₁ in this example.

As also shown in step 84 of FIG. 6A, after the step 82 has been repeatedfor each segment S₂–S_(M), the segment penalties for each resource arerespectively added together amongst the segments S₁–S_(M) to obtain atotal file contention penalty for each resource due to pre-assigning thecandidate file F to the process BP₁. In one aspect of the method of FIG.6A, the total file contention penalties for the candidate file F as afunction of pre-assigned backup process are compiled in a data filepre-assignment contention penalty table, as shown, for example, in FIG.10. Hence, at the completion of the step 84 in FIG. 6A, a respective rowof the table shown in FIG. 10 is filled in. In the current example, therow for backup process BP₁ in a table similar to that shown in FIG. 10for the candidate file F would be filled in with the respective totalcontention penalties (in units of time) for the resources R₁, R₂, and R₃(i.e., disk drive, disk controller, and front-end controller). Again, itshould be appreciated that a table similar to that of FIG. 10 iscompiled for each data file of the target data. Additionally, it shouldbe appreciated that, as with the tables of FIGS. 8 and 9, the table ofFIG. 10 is shown for purposes of illustration only, and that othermanners of compiling and maintaining contention penalty information arepossible, as the invention is not limited to the particular techniqueshown.

After the row of resource contention penalties is completed in FIG. 10for the pre-assignment of the candidate file F to the backup processBP₁, step 86 of FIG. 6A queries if there are any more backup processesspecified for the backup operation. If so, in step 88, the next backupprocess is indicated and the method returns to the step 80, at whichpoint the candidate file F is pre-assigned to the next backup process.The steps 82, 84, 86, and 88 are repeated for each remaining backupprocess, with one row of the table of FIG. 10 being filled in for eachpre-assignment of the candidate file F to a respective backup processwith each iteration of the step 84. Accordingly, after the candidatefile F has been pre-assigned to each backup process, the step 86 passesthe method on to step 90 of FIG. 6A, which indicates that thepre-assignment resource contention penalty table of FIG. 10 for thecandidate file F is complete. The method then proceeds to the step 92 inFIG. 6B.

In step 92, the method selects one or more backup processes from thetable of FIG. 10 for the candidate file F having the lowest contentionpenalty for the resource R₁, which in this example corresponds to a diskdrive penalty. As discussed above, for a primary storage system similarto that shown in FIG. 3, disk drive contention is the most significantfactor leading to performance degradation of a backup operationemploying multiple backup processes. By first seeking to select the oneor more backup processes having the lowest disk drive penalty, themethod of FIGS. 6A and 6B in some sense “weighs” the importance of diskdrive contention by assigning a higher priority to disk drive contentionthan the other resources. As discussed below, the other resourcecontention penalties are considered by the method as secondary factors.Accordingly, it should be appreciated that in other embodiments of theinvention in which a method similar to that of FIGS. 6A and 6B isimplemented with a system having different resources, the availableresources in the computer environment used to access target data may bearranged in terms of their priority with respect to performancedegradation. In this manner, methods similar to that of FIGS. 6A and 6Baccording to other embodiments of the invention may be genericallyapplied to any number and variety of resources.

Additionally, it should be appreciated that as a result of thepre-assignment procedure for the candidate file F discussed above,multiple backup processes may tie for having the lowest contentionpenalty for a given resource. Accordingly, more than one backup processlisted in table of FIG. 10 may have the lowest R₁ penalty.

Once one or more backup processes having the lowest R₁ (e.g., diskdrive) penalty are selected in step 92, the method proceeds in thefollowing steps by considering the relative sizes of the data thatalready has been actually assigned to the respective backup processes(i.e., the respective data loads of the processes). Some of the steps inFIG. 6B illustrate merely one possible technique for considering backupprocess data loads, and it should be appreciated that the invention isnot limited to the specific technique illustrated in FIG. 6B. Ingeneral, according to one aspect of the invention, one goal of assigningthe data files of the target data to the backup processes in aparticular manner is not only to reduce resource contention, but also tokeep the respective data loads of the processes approximately equal, sothat each process completes backing up its respective portion of thetarget data in approximately the same amount of time (i.e., one or moreprocesses do not continue to backup well after others have completedbacking up data). The invention may accomplish such a goal using avariety of techniques, one of which is shown for example in FIG. 6B anddiscussed below.

In particular, in step 94 of FIG. 6B, the elapsed time of each selectedbackup process having the lowest contention penalty for the resource R₁is compared to an elapsed time of the currently shortest backup process.The elapsed time of each selected backup process is measured at thepoint just before the candidate file F is assigned to the backupprocess. The difference in elapsed time between each selected backupprocess and the currently shortest backup process may be expressed asΔt=t _(BP selected) −t _(BP shortest).In step 96, the elapsed time difference Δ t for each selected backupprocess is compared to the backup time t_(F) for the candidate data fileF, as given by the expressionΔt<K ₁ ×t _(F).

The effect of step 96 essentially is to maintain relatively balancedloads between the various backup processes of a backup operation. Forexample, the step 96 essentially asks “if the candidate file F isassigned to one of the selected backup processes having the lowest diskdrive penalty, will this selected backup process be substantially longerthan the currently shortest backup process?” In order to define what“substantially longer” actually means in terms of time, the constant K₁may be selected to be any value (e.g., between 0 and 1), and may beempirically determined for a particular backup storage system. In oneembodiment, the constant K₁ is 0.5; hence, in this embodiment, the step96 tests whether the elapsed time difference Δ t for each selectedbackup process is less than 1/2 of the backup time t_(F) for thecandidate file F.

In step 96 of FIG. 6B, if the elapsed time difference Δ t for at leastone of the selected backup processes is not less than the backup timet_(F) for the candidate file F multiplied by the constant K₁, the methodproceeds to step 98, in which one or more other backup processes havingthe next highest R₁ (e.g., disk drive) penalty are selected from thetable of FIG. 10. The method then returns to the step 94 to againcompare the elapsed time of each new selected backup process to theelapsed time of the currently shortest backup process. The methodrepeats the loop of steps 96, 98, and 94 in FIG. 6B until the query ofstep 96 is satisfied; namely, until the elapsed time difference Δ t forat least one selected backup process is less than the backup time t_(F)for the candidate file F multiplied by the constant K₁. When thiscondition is satisfied, the method proceeds to step 100.

In step 100, the method asks if there is more than one selected backupprocess which satisfied the query of the step 96. If there is only oneselected backup process that satisfied the query, the method proceeds tostep 102, where the candidate file F is actually assigned to theselected backup process. As a result of this assignment, one or morerows of the table shown in FIG. 9 for the selected backup process arefilled in, each row corresponding to one segment of the candidate file Fand identifying the resources used to access the segment. Additionally,one row of the backup process execution table shown in FIG. 11,identifying the candidate data file, is filled in.

In step 104 of FIG. 6B, the method queries if there are more target datafiles. If not, the method proceeds to step 110, wherein the backupoperation is initiated and performed using the backup process executiontable shown in FIG. 11, after which the method ends in step 112.

With reference again to the step 104 in FIG. 6B, if there are moretarget data files after the candidate data file F is assigned to abackup process (e.g., the candidate file F is not F_(smallest) in theembodiment where the target data files are processed in order fromlargest to smallest), the method proceeds to step 106, in which the nextdata file (e.g., the next smallest) is selected. The method thenreturns, via the links 108 (FIG. 6B) and 122 (FIG. 6A), to the step 78,at which point the new (e.g., next smallest) candidate data file ispre-assigned to the first backup process BP₁. Thereafter, the methodperforms the pre-assignment procedure described above for the newcandidate data file.

With reference again to the step 100 in FIG. 6B, if more than oneselected backup process satisfies the query in the step 96(Δt<K₁×t_(F)?), the method proceeds to step 114, which examines thecontention penalties corresponding to the other resources R₂ (e.g., diskcontroller) and R₃ (e.g., front-end controller) listed in the table ofFIG. 10 for each selected backup processes that satisfied the query inthe step 96. In particular, as illustrated in step 114, the methodcalculates an access path collision (APC) penalty for each selectedbackup process, as given by the expressionAPC=[K ₂ ×R ₃ Penalty]+[(1−K ₂)×R ₂ Penalty],where the weighing factor K₂ may be any value between 0 and 1 and, aswith the constant K₁, may be empirically determined. In one embodiment,the weighing factor K₂ is selected to be 0.5. Hence, in this embodiment,each of the contention penalties for the resources R₂ and R₃ are“weighed” equally.

In step 116 of FIG. 6B, the method selects the one or more backupprocess with the lowest access path collision (APC) penalty, and in step118 asks if there is more than one selected backup process having thelowest APC penalty. If only one backup process has the lowest APCpenalty, the method proceeds to step 102, where the candidate file F isassigned to the selected backup process. If, however, more than onebackup process has the same APC penalty, in step 120 of FIG. 6B themethod selects the one of the backup processes sharing the lowest APCpenalty that has the lowest elapsed time difference Δt (i.e. theshortest backup process having the lowest APC), after which the methodproceeds to step 102, where the candidate file F is assigned to theselected backup process.

It should be appreciated that the use and derivation of the APC penaltydiscussed above in connection with the step 114 is merely one example ofa technique for weighing the relative importance of particularresources, and that the invention is not limited to this particulartechnique. For example, alternatively, for each selected backup processthat satisfies the query in the step 96, methods according to otherembodiments of the invention may examine only the R₂ penalties (or onlythe R₃ penalties) individually and select the backup process having thelowest individual R₂ (or R₃) penalty, as opposed to examining analgebraic combination of the R₂ and R₃ penalties (as is the case for theAPC penalty). Furthermore, if two or more backup processes have the sameR₂ (or R₃) penalties, such alternative methods may further examine anindividual penalty for yet another resource, and so on.

Additionally, the embodiment of the invention that prioritizes certainresources is not limited to the implementation discussed above, whereindecisions are made based on one resource (R₁ in the example above), andthen decisions are made for other resources only if there are multipleprocesses equally efficient based on the highest priority resource. Forexample, various mathematical combinations (e.g., algebraicrelationships) of contention penalties for various resources can beevaluated to select a backup process to which the candidate file F isassigned. In this manner, a number of different weighing factors otherthan, or in addition to, the factor K₂ may be employed to implement aparticular weighing scheme for the resources. For example, a particularbackup process to which the candidate file F is assigned may be selectedbased on some criterion for evaluating an expression for an “aggregate”penalty AP having the formAP=[K ₃ ×R ₁ Penalty]+[K ₄ ×R ₂ Penalty]+[K ₅ ×R ₃ Penalty]+ . . . ,where the weighing factors K_(3, 4, 5), . . . each may have a variety ofvalues (e.g., between 0 and 1) to implement a particular weighing orpriority scheme for the various resources R_(1, 2, 3, . . .)

In view of the foregoing, it should be appreciated that although themethod of FIGS. 6A and 6B is directed to assigning data files to backupprocesses based on potential contentions between three particularresources (i.e., disk drives, disk controllers, and front-endcontrollers, as shown in FIG. 3), the method outlined in FIGS. 6A and 6Bmay be adapted for use with any number and variety of computerenvironment resources that are used to access target data during abackup operation. In addition, according to various embodiments of theinvention discussed above, a method similar to that outlined in FIGS. 6Aand 6B enables consideration of resources in a computer environment interms of their relative significance with respect to potentialperformance degradation of a backup operation by, for example,prioritizing the resources in order of importance.

Furthermore, as discussed above in connection with the method outlinedin FIG. 5, the method of FIGS. 6A and 6B, as well as similar methodsaccording to other embodiments, may be iterated a number of times usingdifferent orders of consideration for the candidate data files of thetarget data to compile a number of tables similar to FIG. 11, each tablehaving a particular arrangement of target data file assignments amongthe backup processes. Once a number of such tables are compiled, themethod may determine if one particular arrangement of target data fileassignments results in a shorter completion time for a backup operationthan would other arrangements. In this manner, the method of FIGS. 6Aand 6B, as well as similar methods, may be iterated a number of times tooptimize the assignment of target data files to backup processes so asto insure a shortest completion time for the backup operation.

According to yet another embodiment of the invention, the method ofFIGS. 6A and 6B may be adapted to consider the possibility that each ofthe backup processes BP₁–BP_(N) may not execute at the same data rate r.In practice, since it may not be entirely possible to eliminate resourcecontention amongst the backup processes BP₁–BP_(N), from time to timethe backup processes may not execute at the data rate r due to resourcecontentions.

In view of the foregoing, FIG. 12 illustrates an example of a methodaccording to one embodiment of the invention that takes intoconsideration the possibility of different data rates r amongst thebackup processes BP₁–BP_(N). FIG. 12 is a drawing of a graph similar tothat shown in FIG. 7, illustrating a “snapshot” view of the backupprocesses BP₁–BP_(N) at some point during a data file pre-assignmentprocedure, as discussed above in connection with FIG. 5 and FIGS. 6A and6B. In particular, FIG. 12 illustrates two “guard bands” 200A and 200Brespectively positioned above and below the segment S₁ of the candidatefile F which is pre-assigned to the backup process BP₁. FIG. 12 alsoshows that each of the guard bands 200A and 200B has a respectiveduration GB_(A) and GB_(B).

According to one aspect of this embodiment, the durations GB_(A) andGB_(B) of the guard bands 200A and 200B are related to the duration D₁of the segment S₁. For example, each of the durations GB_(A) and GB_(B)may be approximately 10% of the duration D₁. It should be appreciated,however, that the foregoing example of guard band durations is forpurposes of illustration only, and that the respective durations GB_(A)and GB_(B) of the guard bands 200A and 200B may be the same or differentand, may have a variety of values which may or may not be related to theduration D₁ of the segment S₁.

According to this embodiment, the step 82 in the method of FIGS. 6A and6B (shown particularly in FIG. 6A) is modified such that for eachsegment of a candidate data file F, any resources (R₁, R₂, R₃ . . . )used to access the segment are compared with resources used in all otherbackup processes during a time interval defined by the duration of thesegment plus an additional time interval corresponding to at least one,if not two, guard bands (e.g., one preceding and one following eachsegment). In particular, FIG. 12 shows that the guard band 200B precedesthe segment S₁ and the guard band 200A follows the segment S₁ for thecandidate data file F. In this example, resources are compared amongstthe backup processes BP₁–BP_(N) for the segment S₁ during the timeinterval between (t₁−GB_(B)) to (t₁+D₁+GB_(A)), as indicated on theright vertical axis of FIG. 12. By comparing resources during a timeinterval that is longer than the actual duration of the segment underconsideration, the method adds an extra measure of security againstvariations in data rate r amongst the backup processes BP₁–BP_(N) thatmay lead to inaccurate resource contention determinations if a constantdata rate r is assumed for each backup process.

As discussed above, the guard band durations GB_(B) and GB_(A) shown inFIG. 12 may have a number of values and may be empirically determinedfor a particular backup storage system. Additionally, while FIG. 12illustrates guard bands preceding and following the segment S₁ of thecandidate file F, it should be appreciated that similar guard bands maybe identified for the segment S₂ of the candidate file F. Furthermore,as discussed above, the concept of guard bands may be extended to anysegment S₁–S_(M) of any candidate file F under consideration.

The use of guard bands as shown in FIG. 12 to account for potentialdifferences in the actual execution rates of the backup processesreduces processing time when employed, for example, with the methodsoutlined in FIG. 5 and FIGS. 6A and 6B. This reduced processing time isadvantageous when these methods are performed as part of the backupoperation. However, it should be appreciated that the invention is notlimited in this respect, as other techniques according to the presentinvention may be used to account for potential differences in the actualexecution rates of the backup processes. For example, according to oneembodiment, one or more contention penalties calculated for a particularbackup process during the pre-assignment procedure of either of themethods outlined in FIG. 5 and FIGS. 6A and 6B may be added to theelapsed time of a particular backup process to which a candidate file isactually assigned, to account for resource contention. Similarly, anycontention that results in other backup processes as a result of theactual assignment of the candidate file to the particular backup filemay be accounted for in the other backup processes (i.e., contentiontimes may be added to the elapsed times of the other processes). In thismanner, the elapsed time of each backup process would reflect anycontentions for resources amongst the backup processes that would resultduring the backup operation.

As discussed above in connection with FIG. 4, it should be appreciatedthat the contention controller 34 that performs the above-describedaspects of the present invention can be in a backup storage system orelsewhere in a computer system, and can be implemented in numerous ways,such as with dedicated hardware, or using a processor that is programmedusing microcode or software to perform the functions recited above. Inthis respect, it should be appreciated that one embodiment of theinvention is directed to a computer readable medium (e.g., a computermemory, a floppy disk, a compact disk, a tape, etc.) encoded with acomputer program that, when executed on a computer or other processor,performs methods that implement the various embodiments of the inventiondiscussed above. The computer readable medium can be transportable, suchthat the program stored thereon can be loaded onto one or more differentcomputer systems to implement various aspects of the present inventionas discussed above. It should be understood that the term “computerprogram” is used herein in a generic sense to reference any type ofcomputer code that can be employed to program a computer or otherprocessor to implement various aspects of the present invention asdiscussed above.

Having described several embodiments of the invention in detail, variousmodifications and improvements will readily occur to those skilled inthe art. Such modifications and improvements are intended to be withinthe spirit and scope of the invention. Accordingly, the foregoingdescription is by way of example only, and is not intended as limiting.The invention is limited only as defined by the following claims and theequivalents thereto.

1. In a computer environment including at least one data storage areaand at least one backup data storage system coupled to the at least onedata storage area, the at least one backup storage system configured toexecute at least two backup processes in parallel to backup target datastored in the at least one storage area, each backup process to back upa respective portion of the target data, an apparatus comprising: atleast one contention controller, coupled to the at least one backupstorage system, that distributes the respective portions of the targetdata among the at least two backup processes based, at least in part, onan availability of at least one resource in the computer environmentused by the at least one backup storage system to access the targetdata, wherein each backup process is capable of using the at least oneresource, and wherein the at least one contention controller distributesthe respective portions of the target data among the at least two backupprocesses so as to reduce a contention for the at least one resourcebetween the at least two backup processes, wherein the at least twobackup processes include N backup processes, and wherein the at leastone contention controller pre-assigns a first portion of the target datato a first backup process of the N backup processes, wherein the firstportion is to be backed up during a first time period when the N backupprocesses are executed, and wherein the at least one contentioncontroller makes at least one comparison of contention for at least onefirst resource used to access the first portion between the first backupprocess and at least one other backup process of the N backup processesduring the first time period, and generates at least one contentionpenalty for the first backup process based on the at least onecomparison.
 2. The apparatus of claim 1, wherein after the at least onecontention controller generates the at least one contention penalty forthe first backup process, the at least one contention controller furtherpre-assigns the first portion to a second backup process of the N backupprocesses, the at least one contention controller making at least onecomparison of contention for the at least one first resource used toaccess the first portion between the second backup process and at leastone other backup process of the N backup processes during the first timeperiod and generating at least one contention penalty for the secondbackup process based on the at least one comparison.
 3. The apparatus ofclaim 2, wherein the at least one contention controller assigns thefirst portion of the target data to a particular one of the first backupprocess and the second backup process having the lowest at least onecontention penalty.
 4. The apparatus of claim 2, wherein the at leastone contention controller further consecutively pre-assigns the firstportion to each remaining backup process of the N backup processes untilat least one contention penalty is generated for each backup process ofthe N backup processes.
 5. The apparatus of claim 4, wherein the atleast one contention controller assigns the first portion to aparticular backup process of the N backup processes having the lowest atleast one contention penalty.
 6. The apparatus of claim 4, wherein theat least one contention controller assigns the first portion to aparticular backup process of the N backup processes based on at leastone of the at least one contention penalty for each backup process andan amount of the target data already assigned to each backup process. 7.The apparatus of claim 6, wherein the at least one contention controllermakes a second comparison of a first elapsed time of each backup processhaving the lowest at least one contention penalty and a second elapsedtime of a shortest backup process of the N backup processes, the atleast one contention controller assigning the first portion to theparticular backup process based, at least in part, on the secondcomparison.
 8. The apparatus of claim 7, wherein the at least one firstresource includes a plurality of resources, and wherein the at least onecontention controller prioritizes the plurality of resourcesdifferently.
 9. The apparatus of claim 6, wherein the at least onecontention controller pre-assigns each respective portion of the targetdata to each backup process of the N backup processes to generate atleast one contention penalty for each respective portion in eachrespective process, and assigns each respective portion to a respectiveparticular backup process of the N backup processes based on at leastone of the at least one contention penalty for each respective portionin each backup process and an amount of data already assigned to eachbackup process.
 10. In a computer environment including at least onedata storage area and at least one backup data storage system coupled tothe at least one data storage area, the at least one backup storagesystem configured to execute at least two backup processes in parallelto backup target data stored in the at least one storage area, eachbackup process to back up a respective portion of the target data, anapparatus comprising: at least one contention controller, coupled to theat least one backup storage system, that distributes the respectiveportions of the target data among the at least two backup processesbased, at least in part, on an availability of at least one resource inthe computer environment used by the at least one backup storage systemto access the target data, wherein each backup process is capable ofusing the at least one resource, and wherein the at least one contentioncontroller distributes the respective portions of the target data amongthe at least two backup processes so as to reduce a contention for theat least one resource between the at least two backup processes andwherein the at least two backup processes include N backup processes,and wherein the at least one contention controller pre-assigns a firstportion of the target data to a first backup process of the N backupprocesses, wherein the first portion is to be backed up during a firsttime period when the N backup processes are executed, and wherein the atleast one contention controller makes at least one comparison of thecontention for at least one first resource used to access the firstportion between the first backup process and at least one other backupprocess of the N backup processes during a comparison time period thatincludes at least one guard band time period added to the first timeperiod.
 11. In a computer environment including at least one datastorage area and at least one backup data storage system coupled to theat least one data storage area, the at least one backup storage systemconfigured to execute at least two backup processes in parallel tobackup target data stored in the at least one storage area, each backupprocess to back up a respective portion of the target data, a methodcomprising an act of: a) distributing the respective portions of thetarget data among the at least two backup processes based, at least inpart, on an availability of at least one resource in the computerenvironment used by the at least one backup storage system to access thetarget data wherein the at least two backup processes include N backupprocesses, and wherein the act a) includes acts of: b) pre-assigning afirst portion of the target data to a first backup process of the Nbackup processes, the first portion to be backed up during a first timeperiod when the N backup processes are executed; c) making at least onecomparison of contention for at least one first resource used to accessthe first portion between the first backup process and at least oneother backup process of the N backup processes during the first timeperiod; and d) generating at least one contention penalty for the firstbackup process based on the at least one comparison.
 12. The method ofclaim 11, further including acts of: e) pre-assigning the first portionto a second backup process of the N backup processes; f) making at leastone comparison of contention for the at least one first resource used toaccess the first portion between the second backup process and at leastone other backup process of the N backup processes during the first timeperiod; and g) generating at least one contention penalty for the secondbackup process based on the at least one comparison.
 13. The method ofclaim 12, further including an act of: h) assigning the first portion ofthe target data to a particular one of the first backup process and thesecond backup process having the lowest at least one contention penalty.14. The method of claim 12, further including acts of: h) pre-assigningthe first portion to another backup process of the N backup processes;i) making at least one comparison of contention for the at least onefirst resource used to access the first portion between the other backupprocess and at least one backup process different from the other backupprocess during the first time period; j) generating at least onecontention penalty for the other backup process based on the at leastone comparison; and k) repeating acts h), i), and j) until at least onecontention penalty is generated for each backup process of the N backupprocesses.
 15. The method of claim 14, further including an act of: l)assigning the first portion to a particular backup process of the Nbackup processes having the lowest at least one contention penalty. 16.The method of claim 14, further including an act of: l) assigning thefirst portion to a particular backup process of the N backup processesbased which of the N backup processes have the lowest at least onecontention penalty and an amount of data already assigned to each of theN backup processes.
 17. The method of claim 14, wherein the at least onefirst resource includes a plurality of resources, and wherein the methodfurther includes an act of: k) differently prioritizing the plurality ofresources.
 18. A computer readable medium encoded with a program forexecution on a computer in a computer environment including at least onedata storage area and at least one backup data storage system coupled tothe at least one data storage area, the at least one backup storagesystem configured to execute at least two backup processes in parallelto backup target data stored in the at least one storage area, eachbackup process to back up a respective portion of the target data, theprogram, when executed on the computer, performing a method comprisingan act of: a) distributing the respective portions of the target dataamong the at least two backup processes based, at least in part, on anavailability of at least one resource in the computer environment usedby the at least one backup storage system to access the target data,wherein the at least two backup processes includes N backup processes,and wherein the act a) includes acts of: b) pre-assigning a firstportion of the target data to a first backup process of the N backupprocesses, the first portion to be backed up during a first time periodwhen the N backup processes are executed; c) making at least onecomparison of contention for at least one first resource used to accessthe first portion between the first backup process and at least oneother backup process of the N backup processes during the first timeperiod; and d) generating at least one contention penalty for the firstbackup process based on the at least one comparison.
 19. The computerreadable medium of claim 18, wherein the method further includes actsof: e) pre-assigning the first portion to a second backup process of theN backup processes; f) making at least one comparison of contention forthe at least one first resource used to access the first portion betweenthe second backup process and at least one other backup process of the Nbackup processes during the first time period; and g) generating atleast one contention penalty for the second backup process based on theat least one comparison.
 20. The computer readable medium of claim 19,wherein the method further includes an act of: h) assigning the firstportion of the target data to a particular one of the first backupprocess and the second backup process having the lowest at least onecontention penalty.
 21. The computer readable medium of claim 19,wherein the method further includes acts of: h) pre-assigning the firstportion to another backup process of the N backup processes; i) makingat least one comparison of contention for the at least one firstresource used to access the first portion between the other backupprocess and at least one backup process different from the other backupprocess during the first time period; j) generating at least onecontention penalty for the other backup process based on the at leastone comparison; and k) repeating acts h), i), and j) until at least onecontention penalty is generated for each backup process of the N backupprocesses.
 22. The computer readable medium of claim 21, wherein themethod further includes an act of: l) assigning the first portion to aparticular backup process of the N backup processes having the lowest atleast one contention penalty.
 23. The computer readable medium of claim21, wherein the method further includes an act of: l) assigning thefirst portion to a particular backup process of the N backup processesbased on at least one of the at least one contention penalty for eachbackup process and an amount of data already assigned to each backupprocess.
 24. The computer readable medium of claim 23, wherein the atleast one first resource includes a plurality of resources, and whereinthe method further includes an act of: k) differently prioritizing theplurality of resources.